Tracking imaging control device, tracking imaging system, camera, terminal device, tracking imaging method, and tracking imaging program

ABSTRACT

A tracking imaging control device includes a target color ratio calculation unit that calculates a ratio at which pixels with a certain range of hue occupy in the histogram as the target color ratio, with reference to the color of the target, and a tracking control unit that controls the pan and/or tilt operation of the camera to cause the camera to track the target, and the tracking control unit controls the pan and/or tilt operation of the camera on the basis of information on the position of the target detected by the first target detection unit when the target color ratio is equal to or lower than a threshold value, and controls the pan and/or tilt operation of the camera on the basis of information on the position of the target detected by the second target detection unit when the target color ratio exceeds the threshold value.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of PCT InternationalApplication No. PCT/JP2015/083600 filed on Nov. 30, 2015 claimingpriority under 35 U.S.C §119(a) to Japanese Patent Application No.2015-029238 filed on Feb. 18, 2015. Each of the above applications ishereby expressly incorporated by reference, in their entirety, into thepresent application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a tracking imaging control device, atracking imaging system, a camera, a terminal device, a tracking imagingmethod, and a tracking imaging program in which a pan and/or tiltoperation of a camera including a pan and/or tilt function is controlledand imaging is performed while automatically tracking a target.

2. Description of the Related Art

Generally, in tracking imaging, a position of a target is detected froman image captured by a camera, and a pan and/or tilt operation of thecamera is controlled on the basis of information on the detectedposition of the target to track the target. In this tracking imaging, amethod of using color information of a target is known as one method ofdetecting a position of a target from an image (for example,JP2001-169169A). In this method, the color information of the target isacquired in advance, a subject having the same color as the color of thetarget is detected from the image, and the position of the target isdetected from the image.

However, in a method of detecting a position of a target from an imageusing information on the color of the target, there is a problem in thatit is easy to erroneously detect the target if a larger number of colorssimilar to the target are included in a background.

In order to solve such a problem, a method of detecting a subject thatis a candidate for a target from an image, detecting information on thesubject, selecting an optimal method from among a plurality of methodsof obtaining a position of the target on the basis of information on thedetected subject, and detecting the position of the target has beenproposed in

SUMMARY OF THE INVENTION

However, the method of JP2012-85090A has a disadvantage that a load of aprocess is large since it is necessary to detect the information on thesubject that is a sequential target candidate.

The present invention has been made in view of the above circumstances,and an object of the present invention is to provide a tracking imagingcontrol device, a tracking imaging system, a camera, a terminal device,a tracking imaging method, and a tracking imaging program capable ofsimply detecting a position of a target and accurately tracking thetarget.

Means for solving the above problems are as follows.

[1] A tracking imaging control device that controls a pan and/or tiltoperation of a camera including a pan function and/or a tilt function tocause the camera to execute imaging in which the target is tracked, thetracking imaging control device comprising: a target setting unit thatsets the target; a hue histogram creation unit that creates a histogramof hue of a range in which the target is tracked; a target colorinformation acquisition unit that acquires information on the color ofthe target; a first target detection unit that detects a position of thetarget from the image captured by the camera on the basis of theinformation on the color of the target; a second target detection unitthat detects the position of the target from the image captured by thecamera on the basis of information other than the color of the target; atarget color ratio calculation unit that calculates a ratio at whichpixels with a certain range of hue occupy in the histogram as the targetcolor ratio, with reference to the color of the target; and a trackingcontrol unit that controls the pan and/or tilt operation of the cameraon the basis of information on the position of the target detected bythe first target detection unit when the target color ratio is equal toor lower than a threshold value to cause the camera to track the target,and controls the pan and/or tilt operation of the camera on the basis ofinformation on the position of the target detected by the second targetdetection unit when the target color ratio exceeds the threshold valueto cause the camera to track the target.

According to this aspect, the first target detection unit and the secondtarget detection unit are included as means for detecting the positionof the target. The first target detection unit detects a position of thetarget from the image captured by the camera on the basis of informationon the color of the target. The second target detection unit detects theposition of the target from the image captured by the camera on thebasis of information other than the color of the target. The firsttarget detection unit and the second target detection unit areselectively used on the basis of a relationship between the color of thetarget and a color of a background. That is, in a case where thebackground includes a large number of colors of the target andapproximate colors thereof, it is determined that it is difficult todetect the target on the basis of the color, and the target is trackedon the basis of a detection result of the second target detection unit.In other cases, it is determined that it is possible to detect thetarget on the basis of the color, and the target is tracked on the basisof the detection result of the first target detection unit. Whether ornot the background includes a large number of colors of the target andapproximate colors thereof is determined by creating the histogram ofthe hue in the range in which the target is tracked and calculating thetarget color ratio from the histogram. The target color ratio iscalculated as a ratio at which the pixels with a certain range of hueoccupy in the histogram with reference to the color of the target. In acase where the target color ratio exceeds the threshold value, it isdetermined that the background includes a large number of colors of thetarget and the approximate colors, and the target is tracked on thebasis of the detection result of the second target detection unit. Onthe other hand, in a case where the target color ratio is equal to orlower than the threshold value, it is determined that the number ofcolors of the target and the approximate colors is small, and the targetis tracked on the basis of the detection result of the first targetdetection unit. Thus, according to this aspect, the target color ratiois calculated, and results of the first target detection unit and thesecond target detection unit are selectively used on the basis of thecalculated target color ratio. Accordingly, it is possible to simplydetect the position of the target and accurately track the target.

[2] In the tracking imaging control device of [1], the second targetdetection unit detects the position of the target from the imagecaptured by the camera on the basis of information on luminance orbrightness of the target.

According to this aspect, the second target detection unit detects theposition of the target from the image captured by the camera on thebasis of information on luminance or brightness of the target.Accordingly, even in a case where a background includes a larger numberof colors of the target and approximate colors, the position of thetarget can be detected from the image regardless of color information.

[3] In the tracking imaging control device of [1] or [2], the cameraincludes an imaging unit that captures an optical image of a subjectthrough a lens, and a support unit that supports the imaging unit sothat the imaging unit can be panned and/or tilted.

According to this aspect, the camera includes an imaging unit thatcaptures an optical image of a subject through a lens, and a supportunit that supports the imaging unit so that the imaging unit can bepanned and/or tilted. In a case where the target is tracked, the targetis tracked by panning and/or tilting the imaging unit and changing animaging direction (a direction of the optical axis of the lens).

[4] In the tracking imaging control device of [1] or [2], the cameraincludes an imaging unit that captures an optical image of a subjectthrough a fisheye lens; and an image cutout unit that cuts out a portionof the image captured by the imaging unit, and the pan and/or tiltfunction is realized by changing a position at which the image cutoutunit cuts out an image.

According to this aspect, the camera includes the imaging unit thatcaptures an optical image of a subject through a fisheye lens, and theimage cutout unit that cuts out a portion of the image captured by theimaging unit. In a case where the target is tracked, the target istracked by changing the position at which the image cutout unit cuts outan image.

[5] The tracking imaging control device of any one of [1] to [4] furthercomprises a tracking range setting unit that sets a range in which thetarget is tracked, as the tracking range.

According to this aspect, the tracking range setting unit is furtherincluded. The tracking range setting unit sets a range in which thetarget is tracked, as the tracking range. Accordingly, only a necessaryarea can be set as the tracking range, and the target can be efficientlydetected. Further, it is possible to efficiently create the histogram.

[6] In the tracking imaging control device of [5], the tracking rangesetting unit sets the pan and/or tilt movable range of the camera as thetracking range.

According to this aspect, the tracking range setting unit sets the panand/or tilt movable range of the camera as the tracking range.Accordingly, trouble of setting the tracking range can be reduced.

[7] The tracking imaging control device of [6] further comprises amovable range setting unit that sets a pan and/or tilt movable range ofthe camera.

According to this aspect, a movable range setting unit that sets a panand/or tilt movable range of the camera is further comprised.Accordingly, if the pan and/or tilt movable range of the camera is set,the tracking range can be automatically set and trouble of setting thetracking range can be reduced. Further, pan and/or tilt can be performedonly in a necessary area, and it is possible to efficiently track thetarget.

[8] In the tracking imaging control device of any one of [1] to [7], thehue histogram creation unit creates the histogram of the hue of therange in which the target is tracked, on the basis of image dataobtained by imaging an entire range in which the target is tracked usingthe camera.

According to this aspect, the histogram of the hue of the range in whichthe target is tracked is created on the basis of image data obtained byimaging an entire range in which the target is tracked using the camera.

[9] The tracking imaging control device of any one of [1] to [8] furthercomprises a display unit that displays the image captured by the camera;and an input unit that designates a position on the screen of thedisplay unit, and the target setting unit sets a subject at the positiondesignated by the input unit as the target.

According to this aspect, the display unit that displays the imagecaptured by the camera, and the input unit that designates a position onthe screen of the display unit are further comprised, and a subject atthe position designated by the input unit is set as the target.Accordingly, it is possible to simply set the target.

[10] The tracking imaging control device of any one of [1] to [8]further comprises a face detection unit that detects a face of a personfrom the image captured by the camera, and the target setting unit setsthe face of the person detected by the face detection unit as thetarget.

According to this aspect, the face detection unit that detects a face ofa person from the image captured by the camera is further comprised, andthe face of the person detected by the face detection unit is set as thetarget. Accordingly, it is possible to simply set the target.

[11] The tracking imaging control device of any one of [1] to [8]further comprises a moving body detection unit that detects a movingbody from the image captured by the camera, and the target setting unitsets the moving body first detected by the moving body detection unit asthe target.

According to this aspect, the moving body detection unit that detects amoving body from the image captured by the camera is further comprised,and the moving body first detected by the moving body detection unit isset as the target. Accordingly, it is possible to simply set the target.

[12] In the tracking imaging control device of any one of [1] to [11],the hue histogram creation unit divides the range in which the target istracked into a plurality of blocks, and creates the histogram of the huefor each of the blocks, the target color ratio calculation unitcalculates a target color ratio for each block, the target color ratiobeing a ratio at which pixels with a certain range of hue occupy in thehistogram with reference to the color of the target, and the trackingcontrol unit controls the pan and/or tilt operation of the camera on thebasis of the information on the position of the target detected by thefirst target detection unit for the block in which the target colorratio is equal to or lower than a threshold value to cause the camera totrack the target, and controls the pan and/or tilt operation of thecamera on the basis of the information on the position of the targetdetected by the second target detection unit for the block in which thetarget color ratio exceeds the threshold value to cause the camera totrack the target.

According to this aspect, the range in which the target is tracked isdivided into the plurality of blocks, and the target color ratio iscalculated for each block. A result of means for detecting the positionof the target is selectively used for each block. That is, the target istracked on the basis of the information on the position of the targetdetected by the first target detection unit for the block in which thetarget color ratio is equal to or lower than a threshold value, and thetarget is tracked on the basis of the information on the position of thetarget detected by the second target detection unit for the block inwhich the target color ratio exceeds the threshold value. Thus, even ina case where the color of the background changes, the position of thetarget can be appropriately detected.

[13] A tracking imaging system that includes a camera including a panfunction and/or a tilting function, and a terminal device that iscommunicatably connected to the camera and controls a pan and/or tiltoperation of the camera to cause the camera to execute imaging in whichthe target is tracked, the terminal device comprising: a target settingunit that sets the target; a hue histogram creation unit that creates ahistogram of hue of a range in which the target is tracked; a targetcolor information acquisition unit that acquires information on thecolor of the target; a first target detection unit that detects aposition of the target from the image captured by the camera on thebasis of the information on the color of the target; a second targetdetection unit that detects the position of the target from the imagecaptured by the camera on the basis of information other than the colorof the target; a target color ratio calculation unit that calculates aratio at which pixels with a certain range of hue occupy in thehistogram as the target color ratio, with reference to the color of thetarget; and a tracking control unit that controls the pan and/or tiltoperation of the camera on the basis of information on the position ofthe target detected by the first target detection unit when the targetcolor ratio is equal to or lower than a threshold value to cause thecamera to track the target, and controls the pan and/or tilt operationof the camera on the basis of information on the position of the targetdetected by the second target detection unit when the target color ratioexceeds the threshold value to cause the camera to track the target.

According to this aspect, the target color ratio is calculated, andresults of the first target detection unit and the second targetdetection unit are selectively used on the basis of the calculatedtarget color ratio. Accordingly, it is possible to simply detect theposition of the target and accurately track the target.

[14] A camera comprises: an imaging unit that captures an optical imageof a subject through a lens; a support unit that supports the imagingunit so that the imaging unit can be panned and/or tilted; a targetsetting unit that sets the target; a hue histogram creation unit thatcreates a histogram of hue of a range in which the target is tracked; atarget color information acquisition unit that acquires information onthe color of the target; a first target detection unit that detects aposition of the target from the image captured by the imaging unit onthe basis of the information on the color of the target; a second targetdetection unit that detects the position of the target from the imagecaptured by the imaging unit on the basis of information other than thecolor of the target; a target color ratio calculation unit thatcalculates a ratio at which pixels with a certain range of hue occupy inthe histogram as the target color ratio, with reference to the color ofthe target; and a tracking control unit that controls the pan and/ortilt operation of the imaging unit on the basis of information on theposition of the target detected by the first target detection unit whenthe target color ratio is equal to or lower than a threshold value tocause the target to be tracked, and controls the pan and/or tiltoperation of the imaging unit on the basis of information on theposition of the target detected by the second target detection unit whenthe target color ratio exceeds the threshold value to cause the targetto be tracked.

According to this aspect, the target color ratio is calculated, andresults of the first target detection unit and the second targetdetection unit are selectively used on the basis of the calculatedtarget color ratio. Accordingly, it is possible to simply detect theposition of the target and accurately track the target.

[15] A camera comprises: an imaging unit that captures an optical imageof a subject through a fisheye lens; an image cutout unit that cuts outa portion of an image captured by the imaging unit; a target settingunit that sets the target; a hue histogram creation unit that creates ahistogram of hue of a range in which the target is tracked; a targetcolor information acquisition unit that acquires information on thecolor of the target; a first target detection unit that detects aposition of the target from the image captured by the imaging unit onthe basis of the information on the color of the target; a second targetdetection unit that detects the position of the target from the imagecaptured by the imaging unit on the basis of information other than thecolor of the target; a target color ratio calculation unit thatcalculates a ratio at which pixels with a certain range of hue occupy inthe histogram as the target color ratio, with reference to the color ofthe target; and a tracking control unit that controls the image cutoutunit on the basis of information on the position of the target detectedby the first target detection unit when the target color ratio is equalto or lower than a threshold value to cause the target to be tracked,and controls the image cutout unit on the basis of information on theposition of the target detected by the second target detection unit whenthe target color ratio exceeds the threshold value to cause the targetto be tracked.

According to this aspect, the target color ratio is calculated, andresults of the first target detection unit and the second targetdetection unit are selectively used on the basis of the calculatedtarget color ratio. Accordingly, it is possible to simply detect theposition of the target and accurately track the target.

[16] A terminal device that is communicatably connected to a cameraincluding a pan function and/or a tilting function and controls a panand/or tilt operation of the camera to cause the camera to executeimaging in which the target is tracked, the terminal device comprising:a target setting unit that sets the target; a hue histogram creationunit that creates a histogram of hue of a range in which the target istracked; a target color information acquisition unit that acquiresinformation on the color of the target; a first target detection unitthat detects a position of the target from the image captured by thecamera on the basis of the information on the color of the target; asecond target detection unit that detects the position of the targetfrom the image captured by the camera on the basis of information otherthan the color of the target; a target color ratio calculation unit thatcalculates a ratio at which pixels with a certain range of hue occupy inthe histogram as the target color ratio, with reference to the color ofthe target; and a tracking control unit that controls the pan and/ortilt operation of the camera on the basis of information on the positionof the target detected by the first target detection unit when thetarget color ratio is equal to or lower than a threshold value to causethe camera to track the target, and controls the pan and/or tiltoperation of the camera on the basis of information on the position ofthe target detected by the second target detection unit when the targetcolor ratio exceeds the threshold value to cause the camera to track thetarget.

According to this aspect, the target color ratio is calculated, andresults of the first target detection unit and the second targetdetection unit are selectively used on the basis of the calculatedtarget color ratio. Accordingly, it is possible to simply detect theposition of the target and accurately track the target.

[17] A tracking imaging method of controlling a pan and/or tiltoperation of a camera including a pan function and/or a tilt function tocause the camera to execute imaging in which the target is tracked, thetracking imaging method comprising steps of: setting the target;creating a histogram of hue of a range in which the target is tracked;acquiring information on color of the target; calculating a ratio atwhich pixels with a certain range of hue occupy in the histogram as thetarget color ratio, with reference to the color of the target; anddetecting a position of the target from the image captured by the cameraon the basis of information on the color of the target when the targetcolor ratio is equal to or lower than a threshold value, controlling thepan and/or tilt operation of the camera on the basis of information onthe detected position of the target to cause the camera to track thetarget, and detecting the position of the target from the image capturedby the camera on the basis of information other than the color of thetarget when the target color ratio exceeds the threshold value, andcontrolling the pan and/or tilt operation of the camera on the basis ofinformation on the detected position of the target to cause the camerato track the target.

According to this aspect, the target color ratio is calculated, and amethod of detecting the position of the target is switched on the basisof the calculated target color ratio. Accordingly, it is possible tosimply detect the position of the target and accurately track thetarget.

[18] A tracking imaging program for controlling a pan and/or tiltoperation of a camera including a pan function and/or a tilt function tocause the camera to execute imaging in which the target is tracked isrecorded, the tracking imaging program causing a computer to realizefunctions of: setting the target; creating a histogram of hue of a rangein which the target is tracked; acquiring information on color of thetarget; detecting a position of the target from the image captured bythe camera on the basis of the information on the color of the target;detecting the position of the target from the image captured by thecamera on the basis of information other than the color of the target;calculating a ratio at which pixels with a certain range of hue occupyin the histogram as the target color ratio, with reference to the colorof the target; and detecting a position of the target from the imagecaptured by the camera on the basis of information on the color of thetarget when the target color ratio is equal to or lower than a thresholdvalue, controlling the pan and/or tilt operation of the camera on thebasis of information on the detected position of the target to cause thecamera to track the target, and detecting the position of the targetfrom the image captured by the camera on the basis of information otherthan the color of the target when the target color ratio exceeds thethreshold value, and controlling the pan and/or tilt operation of thecamera on the basis of information on the detected position of thetarget to cause the camera to track the target, and a computer-readablenon-transitory tangible medium having the tracking imaging programrecorded thereon.

According to this aspect, the target color ratio is calculated, and amethod of detecting the position of the target is switched on the basisof the calculated target color ratio. Accordingly, it is possible tosimply detect the position of the target and accurately track thetarget.

According to the present invention, it is possible to simply detect thetarget and track the target accurately.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system configuration diagram illustrating an embodiment of atracking imaging system.

FIG. 2 is a block diagram illustrating a system configuration of acamera.

FIG. 3 is a block diagram illustrating a system configuration of aterminal device.

FIG. 4 is a block diagram illustrating a system configuration of aterminal device functioning as a tracking imaging control device.

FIG. 5 is a diagram illustrating a screen display example of a displaywhen a target is set.

FIG. 6 is a diagram illustrating an example of a screen display of adisplay when a movable range is set.

FIG. 7 is a diagram illustrating an example of image data in a trackingrange.

FIG. 8 is a diagram illustrating an example of a histogram of hue.

FIG. 9 is a conceptual diagram of a method of calculating a target colorratio.

FIG. 10 is a flowchart illustrating a procedure of a tracking imagingprocess in a tracking imaging system

FIG. 11 is a conceptual diagram in a case where a target is tracked byselectively using a first target detection unit and a second targetdetection unit for each block.

FIG. 12 is a block diagram illustrating a system configuration of acamera that electronically realizes a pan and tilt function.

FIG. 13 is a conceptual diagram of image cutout in an image cutout unit.

FIG. 14 is a diagram illustrating a screen display example of a displayof a terminal device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments for carrying out the presentinvention will be described in detail with reference to the accompanyingdrawings.

<<System Configuration>>

FIG. 1 is a system configuration diagram illustrating an embodiment of atracking imaging system according to the present invention.

As illustrated in FIG. 1, a tracking imaging system 1 according to thisembodiment includes a camera 10 having a pan function and a tiltfunction, and a terminal device 100 that controls an operation of thecamera 10.

<Camera>

As illustrated in FIG. 1, the camera 10 includes an imaging unit 12 thatimages a subject, and a support unit 14 that supports the imaging unit12 so that the imaging unit 12 can be panned and tilted.

The imaging unit 12 includes a lens 16, and an image sensor 20 (see FIG.2) that receives light passing through the lens 16. The lens 16 and theimage sensor are housed in the housing 12A and formed as a unit.

The lens 16 has a focusing function and a zooming function. A focus ofthe lens 16 is adjusted by moving a portion of an optical system backand forth along an optical axis L. Further, zoom is adjusted by moving aportion of the optical system back and forth along the optical axis L.The lens 16 is driven by a lens driving unit 16A (see FIG. 2), andfocus, zoom, and iris are adjusted.

The image sensor 20 includes a two-dimensional image sensor such as aCCD image sensor (CCD: Charge Coupled Device) or a CMOS image sensor(CMOS: Complementary Metal Oxide Semiconductor).

The support unit 14 includes an imaging unit support frame 14A thatrotatably supports the imaging unit 12 around a tilt axis T, and agantry 14B that rotatably supports the imaging unit support frame 14Aaround a pan axis P.

The gantry 14B has a substantially rectangular box shape. The gantry 14Bhas a vertical pan axis P at a center, and rotatably supports theimaging unit support frame 14A around the pan axis P. The gantry 14B hasan operation panel 18. Various operation buttons such as a power buttonare included in the operation panel 18. In the camera 10, variousoperations are performed through the operation panel 18.

The imaging unit support frame 14A has a substantially U-shape. Theimaging unit support frame 14A accommodates the imaging unit 12 in agroove-shaped space, and rotatably supports the imaging unit 12 aroundthe tilt axis T. The tilt axis T is set perpendicular to the pan axis P.In the imaging unit 12 supported by the imaging unit support frame 14A,the optical axis L of the lens 16 is orthogonal to the tilt axis T andthe pan axis P.

The imaging unit support frame 14A includes a tilt driving unit 22T (seeFIG. 2) that rotates the imaging unit 12 around the tilt axis T.Further, the gantry 14B includes a pan driving unit 22P that rotates theimaging unit support frame 14A around the pan axis P (see FIG. 2). Thetilt driving unit 22T includes a tilt motor (not illustrated), and theimaging unit 12 is rotated and tilted about the tilt axis T by drivingthe tilt motor. The pan driving unit 22P includes a pan motor (notillustrated), and the imaging unit 12 is rotated and panned about thepan axis P by driving the pan motor.

An angle at which the imaging unit 12 can be panned is, for example,270° (±135°), and an angle at which the imaging unit 12 can be tilted is135° (−45° to +90°).

FIG. 2 is a block diagram illustrating a system configuration of thecamera.

As illustrated in FIG. 2, the camera 10 includes an analog front end(AFE) 24, a camera control unit 30, a memory 50, and a wireless LANcommunication unit (LAN: Local Area Network) 52.

The AFE 24 performs, for example, signal processing such as noiseremoval, signal amplification, or A/D conversion (A/D: Analog/Digital)on the signal (image signal) output from the image sensor 20. A digitalimage signal generated by the AFE 24 is output to the camera controlunit 30.

The camera control unit 30 includes a microcomputer including a centralprocessing unit (CPU) and a memory, and executes a predetermined programto function as an image signal processing unit 32, an imaging controlunit 34, a lens control unit 36, a pan control unit 38P, a tilt controlunit 38T, a communication control unit 40, and a camera operationcontrol unit 42.

The image signal processing unit 32 performs required signal processingon the digital image signal acquired from the AFE 24, to generatedigital image data. For example, the image signal processing unit 32generates digital image data including image data of a luminance signal(Y) and image data of a color difference signal (Cr, Cb).

The imaging control unit 34 controls driving of the image sensor 20 tocontrol imaging of the image sensor 20.

The lens control unit 36 controls the lens driving unit 16A to controloperation of focus, zoom, and an iris of the lens 16.

The pan control unit 38P controls driving of the pan driving unit 22P tocontrol rotation (pan) about the pan axis P of the imaging unit 12.

The tilt control unit 38T controls driving of the tilt driving unit 22Tto control rotation (tilt) about the tilt axis T of the imaging unit 12.

The communication control unit 40 controls the wireless LANcommunication unit 52 to control wireless LAN communication with anexternal device. In the tracking imaging system 1 of this embodiment,communication between the terminal device 100 that is an external deviceis controlled.

The camera operation control unit 42 generally controls an operation ofthe entire camera according to an instruction from the operation panel18 and the terminal device 100.

The memory 50 functions as a storage unit for various pieces of data,and data is written and read according to a request from the cameraoperation control unit 42.

The wireless LAN communication unit 52 performs wireless LANcommunication according to a predetermined wireless LAN standard (forexample, IEEE802.11a/b/g/n standard [IEEE: The Institute of Electricaland Electronics Engineers, Inc./US Institute of Electrical andElectronics Engineers]) with a wireless LAN access point or an externaldevice capable of wireless LAN communication, via an antenna 52A.

<Terminal Device>

The terminal device 100 includes a so-called smart phone, and includes adisplay 102, an operation button 103, a speaker 104, a microphone 105(see FIG. 3), and a built-in camera 106, and the like in a rectangularplate-shaped housing 101, as illustrated in FIG. 1.

FIG. 3 is a block diagram illustrating a system configuration of theterminal device.

As illustrated in FIG. 3, the terminal device 100 includes a CPU 110that controls an overall operation of the terminal device 100, and has aconfiguration in which, for example, a main memory 114, a nonvolatilememory 116, a mobile communication unit 118, a wireless LANcommunication unit 120, a short-range wireless communication unit 122, adisplay unit 124, a touch panel input unit 126, a key input unit 128, anaudio processing unit 130, and an image processing unit 132 areconnected to the CPU 110 via a system bus 112.

The CPU 110 reads an operation program (an operating system (OS) and anapplication program operating on the OS), fixed form data, and the likestored in the nonvolatile memory 116, loads these to the main memory114, and executes the operation program, to function as a control unitthat controls an overall operation of the terminal device.

The main memory 114 includes, for example, a random access memory (RAM),and functions as a work memory of the CPU 110.

The nonvolatile memory 116 includes, for example, a flash EEPROM(EEPROM: Electrically Erasable Programmable Read Only Memory), andstores the above-described operation program or various fixed form data.Further, the nonvolatile memory 116 functions as a storage unit of theterminal device 100 and stores various pieces of data.

The mobile communication unit 118 executes transmission and reception ofdata to and from a nearest base station (not illustrated) via an antenna118A on the basis of a third generation mobile communication systemconforming to an IMT-2000 standard (International MobileTelecommunication-2000) and a fourth generation mobile communicationsystem conforming to an IMT-Advance standard (International MobileTelecommunications-Advanced).

The wireless LAN communication unit 120 performs wireless LANcommunication according to a predetermined wireless LAN communicationstandard (for example, IEEE802.11a/b/g/n standards) with a wireless LANaccess point or an external device capable of wireless LANcommunication, via an antenna 120A.

The short-range wireless communication unit 122 executes transmissionand reception of data to and from a device conforming to anotherBluetooth (registered trademark) standard that is, for example, in arange of (within a radius of about 10 m) of Class 2 via the antenna122A.

The display unit 124 includes a color liquid crystal panel constitutingthe display 102, and a driving circuit therefor, and displays variousimages.

The touch panel input unit 126 is an example of an input unit. The touchpanel input unit 126 is integrally formed with the display 102 using atransparent electrode, and generates and outputs two-dimensionalposition coordinate information corresponding to a touch operation ofthe user.

The key input unit 128 includes a plurality of key switches includingthe operation button 103 included in the housing 101 of the terminaldevice 100, and a driving circuit therefor.

The audio processing unit 130 converts digital audio data provided viathe system bus 112 into an analog signal and outputs the analog signalfrom the speaker 104. Further, the audio processing unit 130 samples theanalog sound signal input from the microphone 105 into digital data andoutputs the digital data.

The image processing unit 132 converts an analog image signal outputfrom the built-in camera 106 including a lens and an image sensor into adigital image signal, performs required signal processing on the digitalimage signal, and outputs a resultant image signal.

<Tracking Imaging Control Device>

In the tracking imaging system 1 of this embodiment, the CPU 110 of theterminal device 100 executes a predetermined tracking imaging program,and the terminal device 100 functions as a tracking imaging controldevice 200.

FIG. 4 is a block diagram illustrating a system configuration of aterminal device functioning as a tracking imaging control device.

A tracking imaging control device 200 includes a target setting unit210, a tracking range setting unit 212, a movable range setting unit214, a hue histogram creation unit 216, a target color informationacquisition unit 218, a first target detection unit 220, a second targetdetection unit 222, a target color ratio calculation unit 224, and atracking control unit 226.

The target setting unit 210 sets a target, that is, a subject that is atracking target. The target setting unit 210 displays an image capturedby the camera 10 on the display 102 and sets a subject touched by theuser on the screen, as the target.

FIG. 5 is a diagram illustrating a screen display example of the displaywhen the target is set.

As illustrated in FIG. 5, when the target is set, the image captured bythe camera 10 is displayed on the display 102. The target setting unit210 acquires image data from the camera 10 and causes the image data tobe displayed on the display 102.

The user confirms a screen display of the display 102, and touches asubject that is the tracking target on the screen. The target settingunit 210 sets a rectangular tracking frame F around a touch position onthe basis of an output from the touch panel input unit 126. The trackingframe F is superimposed on the image and displayed on the display 102.The subject in the tracking frame F is set as the target.

The tracking range setting unit 212 sets a range in which the target istracked (tracking range). The tracking range is set as the pan and tiltmovable range of the camera 10. Therefore, in a case where the pan andtilt movable range is not limited, the entire pan and tilt movable rangeis the tracking range.

The movable range setting unit 214 sets the pan and tilt movable rangeof the camera 10. The movable range setting unit 214 receives adesignation of the pan and tilt movable range from the user, and setsthe pan and tilt movable range. The pan and tilt movable range is set bydetermining a moving end in a positive direction of rotation and amoving end in a negative direction of the rotation. This setting isperformed by actually panning and tilting the camera 10.

FIG. 6 is a diagram illustrating a screen display example of the displaywhen the movable range is set.

As illustrated in FIG. 6, when the pan and tilt movable range is set, animage being captured is displayed in a live view on the display 102. Anarrow P(+) directing panning in a positive direction, an arrowP(−)directing panning in a negative direction, and an arrow T(+)directing tilting in a positive direction, and an arrow T(−)directingtilting in a negative direction are displayed to be superimposed on theimage of the live view on the screen of the display 102.

When the arrow P(+) is touched, the camera 10 is instructed to be pannedin the positive direction, and when the arrow P(−) is touched, thecamera 10 is instructed to be panned in the negative direction. Further,when the arrow T(+) is touched, the camera 10 is instructed to be tiltedin the positive direction, and when the arrow T(−) is touched, thecamera 10 is instructed to be tilted in the negative direction. Themovable range setting unit 214 outputs a pan and tilt instruction to thecamera 10 according to the output from the touch panel input unit 126.

The user instructs the camera 10 to panned and tilted while confirmingthe display of the display 102 and determines a moving end of therotation in a positive direction of the pan, a moving end of therotation in a negative direction of the pan, a moving end of therotation in a positive direction of the tilt, and a moving end of therotation in a negative direction of the tilt. The movable range settingunit 214 sets the pan and tilt movable range on the basis of thedesignated moving end of the rotation.

If the pan and tilt movable range is set, the tracking range settingunit 212 sets the set pan and tilt movable range as the tracking range.

The hue histogram creation unit 216 acquires the image data of thetracking range and creates a histogram of the hue of the tracking range.

FIG. 7 is a diagram illustrating an example of the image data of thetracking range. FIG. 8 is a diagram illustrating an example of ahistogram of the hue.

As illustrated in FIG. 7, the hue histogram creation unit 216 acquiresthe image data of the entire tracking range, and creates the histogramof the hue. Thus, the hue histogram creation unit 216 causes the camera10 to be panned and tilted in the movable range, and acquires the imagedata of the entire tracking range.

The histogram of the hue is represented as a distribution of the numberof pixels for each color value using a hue value as a horizontal axisand a hue value as a vertical axis, as illustrated in FIG. 8.

Data of the created histogram of the tracking range is stored in themain memory 114.

The target color information acquisition unit 218 acquires informationon the color of the target. The target color information acquisitionunit 218 creates the data of the histogram of the hue of the target fromthe image data when the target is selected, and acquires the informationon the color of the target. The data of the histogram of the hue of thetarget is acquired by creating a histogram of the hue of the image inthe tracking frame F. The target color information acquisition unit 218detects a hue value with the greatest pixel value from the data of thecreated histogram of the hue of the target and obtains a hue value thatis the color of the target. Information on data of the created histogramof the hue of the target, and the hue value of the target is stored inthe main memory 114 as the target color information.

The first target detection unit 220 detects the position of the targetfrom the image captured by the camera 10 on the basis of the targetcolor information acquired by the target color information acquisitionunit 218. For the detection of the position of the target using thecolor information, a known technology is used. Hereinafter, a method ofdetecting the position of the target using the information on the colorwill be briefly described.

First, image data of one frame is acquired from the camera 10. Thisimage data is first image data. Then, after a predetermined time haselapsed, the image data of one frame is acquired from the camera 10,similar to the first image data. This image is second image data. Then,a difference between the first image data and the second image data isobtained. The obtained image data is difference image data. Then, thedifference image data is binarized. Accordingly, ideally, for only thepixels of the moving body, one piece of image data is generated. Then,each subject regarded as being integral is labeled on the basis of thebinarized difference image data. Then, an area of the labeled subject isobtained and compared with a threshold value. Then, only the subjectlarger than the threshold value is selected. Accordingly, the subjectsmaller than the threshold value or the subject with a small motion isexcluded. Then, a first-order moment is obtained for each selectedsubject, and a centroid position of each selected subject is obtained.This centroid position, for example, is represented by vertical andhorizontal coordinate values assumed on the screen. Then, for the pixelrange of the selected subject, a histogram of the hue is created fromthe second image data. One subject closest to the histogram of the hueof the target is selected. The selected subject is recognized as thetarget, and a centroid position thereof is recognized as the position ofthe target.

Thus, the first target detection unit 220 detects the position of thetarget from the image captured by the camera 10 on the basis of thetarget color information acquired by the target color informationacquisition unit 218.

The second target detection unit 222 detects the position of the targetfrom the image captured by the camera 10 on the basis of informationother than the target color. In this embodiment, the position of thetarget is detected using known block matching using template. In theblock matching, a motion vector of the target is obtained using atemplate among a plurality of pieces of image data obtained in timeseries, to obtain the position of the target. In this case, for example,the position of the target is obtained using the image in the settracking frame as a template image.

The target color ratio calculation unit 224 calculates a target colorratio X. The target color ratio X is a ratio at which color similar tothe target is included in the background. The target color ratio X iscalculated as follows.

FIG. 9 is a conceptual diagram of a method of calculating the targetcolor ratio.

The target color ratio X is calculated as a ratio at which the pixelswith hue similar to the target occupy in the entirety in the histogramof the hue of the tracking range. A range of the hue similar to thetarget is set as a certain range (TH±α/2°) if a hue value of the targetis TH°. That is, the range of the hue is set from a range of TH−α/2° toTH+α/2°. α is a range in which the hue is recognized as similar hue andis, for example, is 15°. In this case, the range of TH±7.5° is a rangeof hue similar to the target.

The target color ratio calculation unit 224 calculates a ratio at whichthe number of pixels in a range in which the hue value is TH±α/2° occursin the number of pixels of the entire tracking range in the histogram ofthe hue of the tracking range and calculates the target color ratio X.Therefore, when the target color ratio calculation unit 224 calculatesthe target color ratio X, the target color ratio calculation unit 224acquires the histogram data of the hue of the tracking range from thehue histogram creation unit 216 and acquires information on the colorvalue of the target from the target color information acquisition unit218 to calculate the target color ratio X. The calculated target colorratio X is stored in the main memory 114.

The tracking control unit 226 controls the pan and tilt operations ofthe camera 10 to cause the camera 10 to track the target on the basis ofthe position information of the target detected by the first targetdetection unit 220 and the second target detection unit 222. In thisembodiment, the camera 10 is panned and/or tilted so that the target isimaged at a center of the screen. Accordingly, the tracking control unit226 calculates a rotation angle in the panning direction and a rotationangle in the tilt direction required to cause the target to be locatedat the center of the screen on the basis of the position information ofthe target, and outputs the rotation angles to the camera 10.

Incidentally, in the tracking imaging system 1 of this embodiment, thefirst target detection unit 220 and the second target detection unit 222are included as means for detecting the position of the target from theimage captured by the camera 10. The tracking control unit 226selectively uses the first target detection unit 220 and the secondtarget detection unit 222 according to the target color ratio Xcalculated by the target color ratio calculation unit 224. That is, in acase where the target color ratio X calculated by the target color ratiocalculation unit 224 is equal to or lower than a threshold value, thefirst target detection unit 220 is used to detect the target, and in acase where the target color ratio X exceeds the threshold value, thesecond target detection unit 222 is used to detect the target. The casein which the target color ratio X is equal to or lower than thethreshold value is a case where the ratio at which the color similar tothe target is included in the background is low. Therefore, in thiscase, the target is detected using the color information using the firsttarget detection unit 220. On the other hand, the case where the targetcolor ratio X exceeds the threshold value is a case where the ratio atwhich the color similar to the target is included in the background ishigh. Therefore, in this case, the target is detected through blockmatching using the second target detection unit 222. The threshold valueis determined according to whether or not the target can be detectedusing the color information, and an optimal value thereof is determinedfrom a result of a simulation or the like.

The tracking control unit 226 acquires the information on the targetcolor ratio X from the target color ratio calculation unit 224 andcompares the target color ratio X with a threshold value. In a casewhere the target color ratio is equal to or smaller than the thresholdvalue, the tracking control unit 226 controls a pan and/or tiltoperation of the camera 10 on the basis of the position information ofthe target detected by the first target detection unit 220, to cause thecamera 10 to track the target. In a case where the target color ratioexceeds the threshold value, the tracking control unit 226 controls thepan and/or tilt operation of the camera 10 on the basis of the positioninformation of the target detected by the second target detection unit222, to cause the camera 10 to track the target.

<<Tracking Imaging Method>>

FIG. 10 is a flowchart illustrating a processing procedure of trackingimaging in the tracking imaging system of this embodiment.

Tracking imaging is performed by causing the CPU 110 of the terminaldevice 100 to execute the tracking imaging program and causing theterminal device 100 to function as the tracking imaging control device200.

First, the camera 10 and the terminal device 100 are communicatablyconnected to each other. Therefore, communication between the camera 10and the terminal device 100 is established (step S10). By communicatablyconnecting the camera 10 and the terminal device 100 to each other, thecontrol of the camera 10 is enabled on the terminal device 100 side.Further, the image captured by the camera 10 can be displayed on thedisplay 102 of the terminal device 100 or recorded in the nonvolatilememory 116.

Then, setting of the tracking range is performed (step S11). A user canset the pan and tilt movable range of the camera 10 as necessary, to seta tracking range. Information on the set tracking range (pan and tiltmovable range) is stored in the main memory 114.

Then, in order to create the histogram of the hue of the tracking range,image data of the tracking range is acquired (step S12). The terminaldevice 100 causes the camera 10 to be panned and tilted on the basis ofthe information on the set tracking range, and acquires image data ofthe entire tracking range from the camera 10.

If the image data of the entire tracking range is acquired, thehistogram of the hue of the tracking range is created on the basis ofthe image data (step S13). Data of the created histogram is stored inthe main memory 114.

Then, setting of the target is performed (step S14). When the target isset, an image captured over time by the camera 10 is displayed on thedisplay 102 in real time. The user confirms the image displayed on thedisplay 102, and touches and selects the subject that is a target on thescreen. When the target is selected, the image data at the time oftarget selection is stored in the main memory 114. Further, the trackingframe F is set with reference to the touch position and displayed tooverlap the image displayed on the display 102 (see FIG. 5).

Then, the information on the color of the target is acquired (step S15).The terminal device 100 creates a histogram of the hue of the targetfrom the image data at the time of target selection, obtains a hue valueof the target from the created histogram, and acquires the informationon the color of the target. Data of the created histogram of the hue ofthe target and the information on the hue value of the target are storedas the target color information in the main memory 114.

Next, the target color ratio X is calculated on the basis of theinformation on the hue value of the target and the data of the histogramof the hue of the tracking range (step S16). The calculated target colorratio X is stored in the main memory 114.

Then, means for detecting the target is determined on the basis of thecalculated target color ratio X (step S17). That is, the target colorratio X is compared with a threshold value, and it is determined whetheror not the target color ratio X is equal to or smaller than thethreshold value. In a case where the target color ratio X is equal to orsmaller than the threshold value, the first target detection unit 220 isselected, and in a case where the target color ratio X exceeds thethreshold value, the second target detection unit 222 is selected.

If the means for detecting the target is determined, a position of thetarget is detected by the determined means, and the tracking process isperformed on the basis of information on the detected position (stepS18). That is, the position of the target is detected on the basis ofthe image data that are sequentially acquired from the camera 10, andpan and/or tilt of the camera 10 is controlled so that the target isimaged at a center of the screen.

Thereafter, the user instructs the terminal device 100 to record theimage, as necessary, to cause the image captured by the camera 10 to berecorded on the terminal device 100.

Thus, according to the tracking imaging system 1 of this embodiment, aratio at which color similar to the target is included in the backgroundis calculated, and the target is tracked by selectively using means fordetecting the position of the target according to the ratio.Accordingly, it is possible to accurately detect the target and toprevent erroneous tracking. Further, since the histogram of hue of thetracking range is acquired in advance and the means for detecting theposition of the target is determined in advance, it is possible tosimply detect the position of the target without imposing a load to aprocess during a tracking operation.

In a case where a tracking range is an entire pan and tilt movablerange, it is possible to omit the step of setting the tracking range.

Although the target is set after the tracking range is set in the aboveprocessing procedure, the tracking range can be set after the target isset.

MODIFICATION EXAMPLE Modification Example (1) of Tracking Control

Although a configuration in which the first target detection unit 220and the second target detection unit 222 are selectively used on thebasis of the target color ratio X is adopted in the above embodiment, aconfiguration in which results of the first target detection unit 220and the second target detection unit 222 are selectively used on thebasis of the target color ratio X can be adopted. That is, the positiondetection process itself is performed in both of the first targetdetection unit 220 and the second target detection unit 222, and whetheror not to use which of the results is determined on the basis of thetarget color ratio X. In this case, a detection process of the positionof the target in the first target detection unit 220 and a detectionprocess of the position of the target in the second target detectionunit 222 are performed in parallel.

Modification Example (2) of Tracking Control

Although the first target detection unit 220 and the second targetdetection unit 222 are selectively used in relation to the hue of theentire tracking range in the above embodiment, a configuration in whichthe tracking range is divided into a plurality of blocks and the firsttarget detection unit 220 and the second target detection unit 222 isselectively used for each block can be adopted.

FIG. 11 is a conceptual diagram in a case where a target is tracked byselectively using the first target detection unit and the second targetdetection unit for each block.

In this case, the hue histogram creation unit 216 divides the trackingrange into a plurality of blocks and creates a histogram of hue for eachblock. In the example illustrated in FIG. 11, an example in which thetracking range is divided into four blocks B1 to B4 is illustrated. Inthis case, histograms HUE(1) to HUE(4) of the hue of the respectiveblocks B1 to B4 are created individually.

The target color ratio calculation unit 224 calculates the target colorratios X1 to X4 for the respective blocks B1 to B4.

The tracking control unit 226 sets means for detecting a position of thetarget for each block. That is, it is assumed that the block of whichthe target color ratio is equal to or smaller than a threshold value isset using the first target detection unit 220, the block of which thetarget color ratio exceeds the threshold value is set using the secondtarget detection unit 222. Switching between the first target detectionunit 220 and the second target detection unit 222 occurs on the basis ofa current position of the target, the position of the target isdetected, and the pan and/or tilt operation of the camera 10 iscontrolled on the basis of a result of the detection to cause the camera10 to track the target.

The number of blocks in the division can be arbitrarily set by the useror may be automatically set according to a size of the tracking range.Further, the tracking range may be divided only in a pan direction orthe number of divisions in the pan direction and a tilt direction may bechanged.

Modification Example (3) of Tracking Control

Although the pan and/or tilt of the camera 10 is controlled so that thetarget is located at a center of the screen in the above embodiment, thepan and/or tilt of the camera 10 may be controlled so that the target islocated at a position on the screen designated by the user.

Modification Example of Second Target Detection Unit

Although the second target detection unit 222 detects the position ofthe target using the block matching in the above embodiment, the secondtarget detection unit 222 can detect the position of the target using afeature amount other than the color. For example, a configuration inwhich the position of the target is detected from the image captured bythe camera 10 on the basis of information on luminance or brightness ofthe target can be adopted. Specifically, the position of the target canbe detected using an algorithm for object tracking using a knownparticle filter, an algorithm for object tracking using a known gradientmethod, or the like.

Modification Example (1) of Camera

Although the pan and tilt function is realized by a mechanicalconfiguration in the above embodiment, the pan and tilt function can berealized electronically. That is, a portion of the captured image is cutout to generate image data for output, and the pan and/or tilt functionis electronically realized by changing a range for cutting out the imagefor output.

FIG. 12 is a block diagram illustrating a system configuration of acamera that electronically realizes a pan and tilt function.

This camera 300 includes an imaging unit 312 that captures an opticalimage of a subject through a fisheye lens 316, an AFE 324, a cameracontrol unit 330, a memory 350, and a wireless LAN communication unit352.

The imaging unit 312 includes a fisheye lens 316, an image sensor 320that receives light passing through the fisheye lens 316, and a lensdriving unit 316A.

The fisheye lens 316 has a focusing function and is driven by the lensdriving unit 316A so that a focus and an iris are adjusted. The fisheyelens 316 includes, for example, a diagonal fisheye lens.

The image sensor 320 includes a two-dimensional image sensor such as aCCD image sensor or a CMOS image sensor.

The AFE 324 performs, for example, signal processing such as noiseremoval, signal amplification, or A/D conversion on a signal (an imagesignal) output from the image sensor 320. The digital image signalgenerated by the AFE 324 is output to the camera control unit 330.

The memory 350 functions as a storage unit for various pieces of data,and reading and writing of data is performed according to a request froma camera operation control unit 342.

The wireless LAN communication unit 352 performs wireless LANcommunication according to a predetermined wireless LAN standard with awireless LAN access point or an external device capable of wireless LANcommunication, via an antenna 352A.

The camera control unit 330 includes a microcomputer including a CPU anda memory, and functions as an image signal processing unit 332, animaging control unit 334, a lens control unit 336, a communicationcontrol unit 340, a camera operation control unit 342, and an imagecutout unit 344 by executing a predetermined program.

The image signal processing unit 332 performs required signal processingon the digital image signals acquired from the AFE 324 to generatedigital image data. For example, the image signal processing unit 332generates digital image data including image data of a luminance signal(Y) and image data of a color difference signal (Cr, Cb).

The imaging control unit 334 controls driving of the image sensor 320 tocontrol imaging of the image sensor 320.

The lens control unit 336 controls the lens driving unit 316A to controlfocusing of the fisheye lens 316 and an operation of the iris.

The communication control unit 340 controls the wireless LANcommunication unit 352 to control the wireless LAN communication with anexternal device.

The camera operation control unit 342 generally controls the operationof the entire camera according to instructions from the operation unitof the camera 300 and the terminal device (not illustrated).

The image cutout unit 344 acquires the image data generated by the imagesignal processing unit 332 and cuts out a portion of the image togenerate image data for output. The image cutout unit 344 cuts out theimage according to the instruction from the camera operation controlunit 342, to generate image data for output. For example, an image withan instructed aspect ratio is cut out in an instructed size around aninstructed coordinate position to generate image data for output.

FIG. 13 is a conceptual diagram of cutout of an image in the imagecutout unit.

In FIG. 13, an image I1 is an image that is captured by the image sensor320 via the fisheye lens 316. The image cutout unit 344 cuts out aportion of the image I1 and generates an image I2 for output.

The camera 300 outputs the image I2 cut out by the image cutout unit 344as an image for output to the terminal device 100.

FIG. 14 is a diagram illustrating a screen display example of a displayof the terminal device.

As illustrated in FIG. 14, an image I2 cut out from the image I1captured through the fisheye lens 316 is displayed as the image capturedby the camera 300 in the display 102 of the terminal device 100.

Thus, the camera 300 that electronically realizes a pan and tiltfunction is configured to cut out a portion of an actually imaged imageand output image data and configured to be panned and/or tilted bychanging a cutout position.

Although the configuration in which a portion of the image captured bythe single imaging unit is cut out and the image data for output isacquired is adapted in the above example, a configuration in which aplurality of imaging units are included in the camera, images capturedby the plurality of imaging units are combined to generate a singleimage, a portion of the image is cut out, and image data for output isacquired can be adopted. For example, a configuration in which a firstimaging unit that images the front and a second imaging unit that imagesthe rear are included, an image captured by the first imaging unit andan image captured by the second imaging unit are combined to generateone image, a camera capable of imaging 360° in a pan direction isformed, a portion of the image is cut out, and image data for output isacquired can be adopted.

Modification Example (2) of Camera

Although the camera 10 of the above embodiment includes the function ofthe pan and tilt function, the camera may include at least the pan ortilt function. In the case of a camera including only a pan function,tracking of the target is performed only in a pan operation. Similarly,in the case of a camera including only a tilt function, tracking of thetarget is performed only in a tilt operation.

Modification Example of Target Setting Unit

In the above-described embodiment, the image captured by the camera 10is displayed on the display 102 and a subject on the screen touched bythe user is set as the target, but a method of setting the target is notlimited thereto.

For example, a configuration in which a function of automaticallydetecting a face of a person from the image captured by the camera (afunction of the face detection unit) is added as a function of thetracking imaging control device, and the face of the person detectedusing the function is automatically set as the target can be adopted.Accordingly, it is possible to simply set the target.

In this case, the plurality of faces may be detected, but in this case,for example, a configuration in which a result of the detection isdisplayed to the user and a subject is selected as the target can beadopted. Further, a configuration in which the target can beautomatically determined from a size or a position of the detected facecan be adopted. For example, a main subject is determined under adetermination criterion that a face located at a center of the screenseems to be the main subject and a larger face seems to be the mainsubject, and the target is automatically o determined.

Further, for example, a configuration in which a function of detecting amoving body from the image captured by the camera (a function of amoving body detection unit) is added as a function of the trackingimaging control device, and a moving body first detected using thefunction is set as the target can be adopted. Accordingly, it ispossible to simply set the target.

In this case, a plurality of moving bodies may be detected at the sametime, but in this case, a configuration in which a user is caused toselect the subject that is a target can be adopted. Alternatively, aconfiguration in which the target is automatically determined from asize or a position of the detected moving body can be adopted.

Further, although the tracking frame having a predetermined size is seton the basis of touch position information in the above embodiment, theposition and the size of the tracking frame may be adjusted by the user.

Further, a position and a size of the tracking frame may beautomatically adjusted. For example, a moving body may be extracted withreference to the touch position and the tracking frame may be set tosurround the moving body. Alternatively, a face of a person may beextracted with reference to a touch position and the tracking frame maybe set to surround the face.

Further, although the image captured by the camera is displayed on thedisplay of the terminal device in real time and the target is selectedin the above embodiment, a configuration in which a still image iscaptured and displayed on the display and the target is selected can beadopted.

Further, a configuration in which the image of the target is registeredin advance and read to set the target can be adopted.

Modification Example of Calculation of Target Color Ratio

In the above embodiment, when the hue value of the target is TH°, arange of TH±α/2° is used as a range of hues similar to the target, andthe target color ratio is calculated. An example of α is 15°. α set asthe range of the hue similar to the target may be a fixed value or maybe set arbitrarily. Further, a configuration in which the value of α isautomatically set according to the ratio of color similar to the targetincluded in the tracking range may be adopted. For example, thehistogram of the tracking range is analyzed, and when a ratio of the huesimilar to the target is higher, the value of α is set to a small value.That is, when a larger number of colors similar to the target areincluded in the tracking range that is a background, the value of α isset to a small value.

If the α is changed in this way, the target color ratio X is changed anda frequency at which the position of the target is detected using thecolor information is changed. Therefore, in a case where α is changed,the threshold value is changed in conjunction with the change in α sothat the frequency at which the position of the target is detected usingthe color information is changed. That is, in a case where α decreases,the threshold value decreases in conjunction with this, and in a casewhere α increases, the threshold value increases in conjunction withthis. Accordingly, the position detection of the target using the colorinformation and the position detection of the target using informationother than the color can be selectively used.

Modification Example of Method of Setting Tracking Range

In the above embodiment, the pan and tilt movable range is set as thetracking range, a method of setting the tracking range is not limitedthereto. For example, the range in which the user performs imaging canbe set as the tracking range using the pan and tilt function of thecamera 10. In this case, as a pre-setting operation, the user performsimaging in a range that is the tracking range using the pan and tiltfunction of the camera 10. Thus, in a case where the range in which theuser performs imaging is a tracking range, an image of an entiretracking range required for creation of the hue histogram can besimultaneously acquired.

In a case where the pan and tilt movable range is set, a range in whichthe user has performed imaging can be used as the pan and tilt movablerange.

<Camera Including Tracking Imaging Function>

Although the configuration in which the terminal device functions as thetracking imaging control device, and the terminal device detects theposition of the target and controls the pan and tilt of the camera isadopted in the above embodiment, a configuration in which the camera isequipped with the function of the tracking imaging control device, andthe camera detects the position of the target and controls the pan andtilt of the camera can be adopted. In this case, the camera is equippedwith the functions of the target setting unit, the hue histogramcreation unit, the first target detection unit, the second targetdetection unit, the target color ratio calculation unit, and thetracking control unit. These functions can be provided as functions ofthe camera control unit. That is, the microcomputer constituting thecamera control unit can cause the camera control unit to function as thetarget setting unit, the hue histogram creation unit, the first targetdetection unit, the second target detection unit, the target color ratiocalculation unit, and the tracking control unit by executing apredetermined tracking imaging program.

Thus, in a case where the camera is equipped with the function of thetracking imaging control device, the terminal device can be configuredto perform only a display of the image captured by the camera or onlythe display and recording of the image. Alternatively, the terminaldevice can be configured to perform only setting of the target.

Further, in the case where the camera is equipped with the function ofthe tracking imaging control device in this way, the camera can beoperated alone to perform the tracking imaging. In this case, it ispreferable for the camera to include a display unit and a touch panelinput unit.

<Connection Form Between Camera and Terminal Device>

Although the camera and the terminal device are connected wirelesslycommunicatably in the above embodiment, the camera and the terminaldevice may be connected mutually communicatably. Therefore, the cameraand the terminal device may be connected communicatably in a wiredmanner. Further, a communication standard or the like is notparticularly limited. Further, the camera and the terminal device arenot directly connected and, for example, the camera and the terminaldevice may be connected over the Internet.

<Terminal Device>

In the above embodiment, the smart phone is adopted as the terminaldevice, but the form of the terminal device is not particularly limited.Therefore, the terminal device can include a personal computer or atablet computer. Further, the terminal device can include a dedicateddevice.

<Display of Histogram of Hue of Tracking Range>

Although the data of the created histogram of the hue of the trackingrange is used only for calculation of the target color ratio in theabove embodiment, the histogram of the hue of the tracking range may becreated and then the data thereof may be displayed on the display unit.Accordingly, the user can use the data as a judgment material whensetting the target. The data of the histogram, for example, can bedisplayed over the screen at the time of setting of the target.

<Presentation of Detectable Color>

By acquiring the data of the histogram of the hue of the tracking range,it is possible to obtain the color from which the position of the targetcan be detected using the color information in advance. That is, byacquiring the data of the histogram of the hue of the tracking range, itis possible to obtain the hue value at which the target color ratio isequal to or lower than the threshold value from the data. Accordingly,by acquiring the data of the histogram of the hue of the tracking range,the color from which the position of the target can be detected usingthe color information can be obtained in advance. Thus, color from whichthe position of the target can be detected may be obtained from the dataof the histogram of the hue of the tracking range in advance, andinformation on the obtained color may be presented to the user at thetime of setting of the target. For example, the information on the color(hue) from which the position of the target can be detected using thecolor information can be displayed to be superimposed on the screen atthe time of setting of the target. Accordingly, the user can use theinformation as a judgment material when setting the target

EXPLANATION OF REFERENCES

-   1: tracking imaging system-   10: camera-   12: imaging unit-   12A: housing-   14: support unit-   14A: imaging unit support frame-   14B: gantry-   16: lens-   16A: lens driving unit-   18: operation panel-   20: image sensor-   22P: pan driving unit-   22T: tilt driving unit-   30: camera control unit-   32: image signal processing unit-   34: imaging control unit-   36: lens control unit-   38P: pan control unit-   38T: tilt control unit-   40: communication control unit-   42: camera operation control unit-   50: memory-   52: wireless LAN communication unit-   52A: antenna-   100: terminal device-   101: housing-   102: display-   103: operation button-   104: speaker-   105: microphone-   106: built-in camera-   110: CPU-   112: system bus-   114: main memory-   116: nonvolatile memory-   118: mobile communication unit-   118A: antenna-   120: wireless LAN communication unit-   120A: antenna-   122: near field wireless communication unit-   122A: antenna-   124: display unit-   126: touch panel input unit-   128: key input unit-   130: audio processing unit-   132: image processing unit-   200: tracking imaging control device-   210: target setting unit-   212: tracking range setting unit-   214: movable range setting unit-   216: hue histogram creation unit-   218: target color information acquisition unit-   220: first target detection unit-   222: second target detection unit-   224: target color ratio calculation unit-   226: tracking control unit-   300: camera-   312: imaging unit-   316: fisheye lens-   316A: lens driving unit-   320: image sensor-   330: camera control unit-   332: Image signal processing unit-   334: imaging control unit-   336: lens control unit-   340: communication control unit-   342: camera operation control unit-   344: image cutout unit-   350: memory-   352: wireless LAN communication unit-   352A: antenna-   F: range of tracking frame

What is claimed is:
 1. A tracking imaging control device that controls apan and/or tilt operation of a camera including a pan function and/or atilt function to cause the camera to execute imaging in which the targetis tracked, the tracking imaging control device comprising: a targetsetting unit that sets the target; a hue histogram creation unit thatcreates a histogram of hue of a range in which the target is tracked; atarget color information acquisition unit that acquires information onthe color of the target; a first target detection unit that detects aposition of the target from the image captured by the camera on thebasis of the information on the color of the target; a second targetdetection unit that detects the position of the target from the imagecaptured by the camera on the basis of information other than the colorof the target; a target color ratio calculation unit that calculates aratio at which pixels with a certain range of hue occupy in thehistogram as the target color ratio, with reference to the color of thetarget; and a tracking control unit that controls the pan and/or tiltoperation of the camera on the basis of information on the position ofthe target detected by the first target detection unit when the targetcolor ratio is equal to or lower than a threshold value to cause thecamera to track the target, and controls the pan and/or tilt operationof the camera on the basis of information on the position of the targetdetected by the second target detection unit when the target color ratioexceeds the threshold value to cause the camera to track the target. 2.The tracking imaging control device according to claim 1, wherein thesecond target detection unit detects the position of the target from theimage captured by the camera on the basis of information on luminance orbrightness of the target.
 3. The tracking imaging control deviceaccording to claim 1, wherein the camera includes an imaging unit thatcaptures an optical image of a subject through a lens, and a supportunit that supports the imaging unit so that the imaging unit can bepanned and/or tilted.
 4. The tracking imaging control device accordingto claim 1, wherein the camera includes an imaging unit that captures anoptical image of a subject through a fisheye lens; and an image cutoutunit that cuts out a portion of the image captured by the imaging unit,wherein the pan and/or tilt function is realized by changing a positionat which the image cutout unit cuts out an image.
 5. The trackingimaging control device according to claim 1, further comprising: atracking range setting unit that sets a range in which the target istracked, as the tracking range.
 6. The tracking imaging control deviceaccording to claim 5, wherein the tracking range setting unit sets thepan and/or tilt movable range of the camera as the tracking range. 7.The tracking imaging control device according to claim 6, furthercomprising: a movable range setting unit that sets a pan and/or tiltmovable range of the camera.
 8. The tracking imaging control deviceaccording to claim 1, wherein the hue histogram creation unit createsthe histogram of the hue of the range in which the target is tracked, onthe basis of image data obtained by imaging an entire range in which thetarget is tracked using the camera.
 9. The tracking imaging controldevice according to claim 1, further comprising: a display unit thatdisplays the image captured by the camera; and an input unit thatdesignates a position on the screen of the display unit, wherein thetarget setting unit sets a subject at the position designated by theinput unit as the target.
 10. The tracking imaging control deviceaccording to claim 1, further comprising: a face detection unit thatdetects a face of a person from the image captured by the camera,wherein the target setting unit sets the face of the person detected bythe face detection unit as the target.
 11. The tracking imaging controldevice according to claim 1, further comprising: a moving body detectionunit that detects a moving body from the image captured by the camera,wherein the target setting unit sets the moving body first detected bythe moving body detection unit as the target.
 12. The tracking imagingcontrol device according to claim 1, wherein the hue histogram creationunit divides the range in which the target is tracked into a pluralityof blocks, and creates the histogram of the hue for each of the blocks,the target color ratio calculation unit calculates a target color ratiofor each block, the target color ratio being a ratio at which pixelswith a certain range of hue occupy in the histogram with reference tothe color of the target, and the tracking control unit controls the panand/or tilt operation of the camera on the basis of the information onthe position of the target detected by the first target detection unitfor the block in which the target color ratio is equal to or lower thana threshold value to cause the camera to track the target, and controlsthe pan and/or tilt operation of the camera on the basis of theinformation on the position of the target detected by the second targetdetection unit for the block in which the target color ratio exceeds thethreshold value to cause the camera to track the target.
 13. A trackingimaging system that includes a camera including a pan function and/or atilting function, and a terminal device that is communicatably connectedto the camera and controls a pan and/or tilt operation of the camera tocause the camera to execute imaging in which the target is tracked, theterminal device comprising: a target setting unit that sets the target;a hue histogram creation unit that creates a histogram of hue of a rangein which the target is tracked; a target color information acquisitionunit that acquires information on the color of the target; a firsttarget detection unit that detects a position of the target from theimage captured by the camera on the basis of the information on thecolor of the target; a second target detection unit that detects theposition of the target from the image captured by the camera on thebasis of information other than the color of the target; a target colorratio calculation unit that calculates a ratio at which pixels with acertain range of hue occupy in the histogram as the target color ratio,with reference to the color of the target; and a tracking control unitthat controls the pan and/or tilt operation of the camera on the basisof information on the position of the target detected by the firsttarget detection unit when the target color ratio is equal to or lowerthan a threshold value to cause the camera to track the target, andcontrols the pan and/or tilt operation of the camera on the basis ofinformation on the position of the target detected by the second targetdetection unit when the target color ratio exceeds the threshold valueto cause the camera to track the target.
 14. A camera, comprising: animaging unit that captures an optical image of a subject through a lens;a support unit that supports the imaging unit so that the imaging unitcan be panned and/or tilted; a target setting unit that sets the target;a hue histogram creation unit that creates a histogram of hue of a rangein which the target is tracked; a target color information acquisitionunit that acquires information on the color of the target; a firsttarget detection unit that detects a position of the target from theimage captured by the imaging unit on the basis of the information onthe color of the target; a second target detection unit that detects theposition of the target from the image captured by the imaging unit onthe basis of information other than the color of the target; a targetcolor ratio calculation unit that calculates a ratio at which pixelswith a certain range of hue occupy in the histogram as the target colorratio, with reference to the color of the target; and a tracking controlunit that controls the pan and/or tilt operation of the imaging unit onthe basis of information on the position of the target detected by thefirst target detection unit when the target color ratio is equal to orlower than a threshold value to cause the target to be tracked, andcontrols the pan and/or tilt operation of the imaging unit on the basisof information on the position of the target detected by the secondtarget detection unit when the target color ratio exceeds the thresholdvalue to cause the target to be tracked.
 15. A camera, comprising: animaging unit that captures an optical image of a subject through afisheye lens; an image cutout unit that cuts out a portion of an imagecaptured by the imaging unit; a target setting unit that sets thetarget; a hue histogram creation unit that creates a histogram of hue ofa range in which the target is tracked; a target color informationacquisition unit that acquires information on the color of the target; afirst target detection unit that detects a position of the target fromthe image captured by the imaging unit on the basis of the informationon the color of the target; a second target detection unit that detectsthe position of the target from the image captured by the imaging uniton the basis of information other than the color of the target; a targetcolor ratio calculation unit that calculates a ratio at which pixelswith a certain range of hue occupy in the histogram as the target colorratio, with reference to the color of the target; and a tracking controlunit that controls the image cutout unit on the basis of information onthe position of the target detected by the first target detection unitwhen the target color ratio is equal to or lower than a threshold valueto cause the target to be tracked, and controls the image cutout unit onthe basis of information on the position of the target detected by thesecond target detection unit when the target color ratio exceeds thethreshold value to cause the target to be tracked.
 16. A terminal devicethat is communicatably connected to a camera including a pan functionand/or a tilting function and controls a pan and/or tilt operation ofthe camera to cause the camera to execute imaging in which the target istracked, the terminal device comprising: a target setting unit that setsthe target; a hue histogram creation unit that creates a histogram ofhue of a range in which the target is tracked; a target colorinformation acquisition unit that acquires information on the color ofthe target; a first target detection unit that detects a position of thetarget from the image captured by the camera on the basis of theinformation on the color of the target; a second target detection unitthat detects the position of the target from the image captured by thecamera on the basis of information other than the color of the target; atarget color ratio calculation unit that calculates a ratio at whichpixels with a certain range of hue occupy in the histogram as the targetcolor ratio, with reference to the color of the target; and a trackingcontrol unit that controls the pan and/or tilt operation of the cameraon the basis of information on the position of the target detected bythe first target detection unit when the target color ratio is equal toor lower than a threshold value to cause the camera to track the target,and controls the pan and/or tilt operation of the camera on the basis ofinformation on the position of the target detected by the second targetdetection unit when the target color ratio exceeds the threshold valueto cause the camera to track the target.
 17. A tracking imaging methodof controlling a pan and/or tilt operation of a camera including a panfunction and/or a tilt function to cause the camera to execute imagingin which the target is tracked, the tracking imaging method comprisingsteps of: setting the target; creating a histogram of hue of a range inwhich the target is tracked; acquiring information on color of thetarget; calculating a ratio at which pixels with a certain range of hueoccupy in the histogram as the target color ratio, with reference to thecolor of the target; and detecting a position of the target from theimage captured by the camera on the basis of information on the color ofthe target when the target color ratio is equal to or lower than athreshold value, controlling the pan and/or tilt operation of the cameraon the basis of information on the detected position of the target tocause the camera to track the target, and detecting the position of thetarget from the image captured by the camera on the basis of informationother than the color of the target when the target color ratio exceedsthe threshold value, and controlling the pan and/or tilt operation ofthe camera on the basis of information on the detected position of thetarget to cause the camera to track the target.
 18. A computer-readablenon-transitory tangible medium on which a tracking imaging program forcontrolling a pan and/or tilt operation of a camera including a panfunction and/or a tilt function to cause the camera to execute imagingin which the target is tracked is recorded, the tracking imaging programcausing a computer to realize functions of: setting the target; creatinga histogram of hue of a range in which the target is tracked; acquiringinformation on color of the target; detecting a position of the targetfrom the image captured by the camera on the basis of the information onthe color of the target; detecting the position of the target from theimage captured by the camera on the basis of information other than thecolor of the target; calculating a ratio at which pixels with a certainrange of hue occupy in the histogram as the target color ratio, withreference to the color of the target; and detecting a position of thetarget from the image captured by the camera on the basis of informationon the color of the target when the target color ratio is equal to orlower than a threshold value, controlling the pan and/or tilt operationof the camera on the basis of information on the detected position ofthe target to cause the camera to track the target, and detecting theposition of the target from the image captured by the camera on thebasis of information other than the color of the target when the targetcolor ratio exceeds the threshold value, and controlling the pan and/ortilt operation of the camera on the basis of information on the detectedposition of the target to cause the camera to track the target.